We tried it first with just the RNN, and while it worked reasonably well, it had some odd edge cases.
We didn't have enough time to train it again, or generate a better dataset, so I cooked up a simple naive bayesian classifier in Lua, saw that it had comparable success, and averaged the results.
That approach worked excellently.